I/O management: tools and conventions in HySoP

Different tools are available in HySoP to deal with file reading or writing, for:

  • scalar or vector fields, in hdf files,

  • basic text (ascii) files, for example to write the time evolution of a specific variable.

Files descriptors: IOParams

For I/O operations, especially in a parallel context, some specfici parameters must be properly defined. In HySoP, IOParams object is devoted to this task.

Whenever you need to access to a file, you must use IOParams. Most of the time, you can rely on the default behavior and just specify a file name, like this

>>> from hysop import IOParams, IO
>>> iop = IOParams('myfile.h5')
>>> # use iop as input argument for i/o operators
>>> # ...
>>> # or to explicitely set the file type:
>>> iop2 = IOParams('f2.h5', fileformat=IO.HDF5)
>>> iop3 = IOParams('f3.txt', fileformat=IO.ASCII)

In any case, a default path will be set, the file created if needed, an mpi communicator associated with it and so on.

Let us now get more into details about IOParams. Such an object defines uniquely:

  • IOParams.filename, name of the output/input file (relative or absolute)

  • IOParams.filepath: full path to filename. Ignored if filename is absolute.

  • IOParams.frequency: frequency of access to this file (e.g. if equal to N, print or read every N time steps), default = 1

  • IOParams.fileformat: the chosen format (ASCII, HDF …), default=IO.HDF5

  • IOParams.io_leader, the rank (in communicator of the object using IOParams) of the mpi process that will lead the read/write process, default=0.

Filename is the only required parameter, other ones have default values as shown above.

If filename is ‘some_relative_path/name’, then filepath=$PWD/some_relative_path/name.

If filename is ‘just_a_name’, then filepath=default_path/just_a_name where default_path depends on the way you run python (interactivly, ipython …) and is equal to default_path() (see table below). To check your current value, try:

from hysop.tools.io_utils import IO
print IO.default_path()

Type of execution

Default path values

interactive

$PWD/interactive/p1

command line

$PWD/pythonfile_name/pN

N stands for the number of mpi processes used, and $PWD for current directory. For example, if you run in /tmp/hysop:

mpirun -np 4 myscript.py

default path will be /tmp/hysop/myscript/p4

Anyway, since the behavior of default_path strongly depends on your python version, the way you run your scropt and others things, it may be a good idea to explicitely set default path at the beginning of your python main program, using:

from hysop import IO
IO.set_default_path('somewhere')

Notice that even in this case, pN will be appended to ‘somewhere’.

Examples:

from hysop import IOParams
# minimal config
io1 = IOParams('toto.h5')
print io1

io2 = IOParams('/tmp/myexamples/tutu.h5')
io3 = IOParams('tutu.h5', filepath='/tmp/myexamples/', frequency=4)

As usual, check tools/tests/test_io.py for an overview of all behaviors.

About HDF5 files in HySoP

HDF5 is, among other things, a portable file format well fitted and efficient to deal with parallel I/O for high volume of data. In HySoP, the python interface h5py is used.

References :

Here is a short reminder of the basic things you need to know about hdf5 in HySoP.

A dataset is a multidimensionnal array of data, in our case the values of a scalar field (or of a component of a vector field) on a grid. Each dataset may have some attributes (name …)

An hdf file is a container of datasets, with all the associated meta-data. It is thus important to notice that a single hdf file contains all the informations concerning the mesh on which datasets are defined, the name of the fields, ldots and is in some way self-sufficient.

File name convention

In HySoP, hdf files are always named ‘somename.h5’. Each file can contain several datasets but corresponds to one and only one instant of simulation.

When fields are written at several time instants (see for example HDF_Writer) during simulation, hdf files are named ‘somename_XXXXX.h5’ and a ‘somename.xmf’ file is created.

XXXXX is the iteration number which corresponds to the time when the file has been written. The xmf file gathered all h5 files, i.e the complete collection of field values at all time steps and can be post-processed with Paraview or any similar tool.

h5 file can be ‘visualized’ with h5dump or HDFView or uploaded into python (see h5py doc).

Saving fields to HDF files

The operator HDF_Writer is the only way to dump properly a given field values to hdf5 file. One still can use manually the h5py module to dump the components. The provided operator handle the geometry of the domain and time dependant datas.

Let us start with an example

>>> from hysop import Box, Field, Simulation, Problem
>>> from hysop.operators import HDF_Writer
>>> dom = Box(dim=3)
>>> f = Field(dom, name='f1', is_vector=True)
>>> w_op = HDF_Writer(name='dump', variables={f: (33,)*3})
>>> problem = Problem()
>>> _ = problem.insert(w_op)
>>> _ = problem.build()
>>> simu = Simulation(nb_iter=2)
>>> simu.initialize()
>>> problem.solve(simu)

Executing this will create three files named f1*.xmf, f1*_000000.h5 and f1*_000001.h5